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COMBINED STEREOVISION, COLOR 3D DIGITIZING 
AND MOTION CAPTURE SYSTEM 



FIELD OF THE INVENTION 

The present invention relates to a digitizer combining 
5 functions of stereovision, color 3D digitizing and motion 
capture of a target object, a digitizing system using the 
digitizer, and associated digitizing and motion tracking 
methods . 

BACKGROUND 

10 3D digitizing, particularly non-contact optical 3D 

digitizing techniques, have become commercially available 
during recent years. Most of these techniques are based on 
the principle of optical triangulation . Despite the fact that 
' passive optical triangulation (stereovision) has been studied 

15 and used for many years for photogrammetric measurements, the 
active optical triangulation technique (particularly laser 
scanning technique) has gained popularity because of its 
robustness and simplicity to process the obtained data using 
a computer. Most of the systems based on the active optical 

20 triangulation principle were developed for industrial 
applications, such as robotics assembly, robot guidance, 
industrial inspection, reverse engineering, etc. A laser beam 
or a laser stripe is projected onto a 3D surface of an 
object, scattering the laser beam or laser stripe on the 

25 surface. It is measured using a photo-electronic device. A 
signal can be measured indicating the position (usually the 
depth) of the measuring point. In most cases, the basic 
measurements are either a point or a section profile. A 
mechanical or optical scanning device is usually used to 

30 provide a frame of 3D measurement. Laser is a monochromatic 
light source that does not provide full color information. 
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SO, an additional camera and light source are used when a 
color texture is needed. 

A new category of optical color 3D digitizers, such as 
the present applicant • s product line, has been developed. 
5 . These systems use structured white light projection combined 
with a CCD camera allowing for the measurement of 3D geometry 
and color texture of a surface. The projected structured 
light (viewed by a camera from an angle different from the 
light projection) is deformed due to the 3D surface relief 
10. The 3D coordinates of the surface are calculated by analyzing 
the deformation. These kinds of systems are being used in 
computer animation, special effects and in electronic game 
development . 

on the other hand, the passive optical triangulation 
15 (stereovision, for example) is largely used for the purpose 
Of motion capture. The correspondence problem (automatically 
finding one point on the object's surface from two optical 
sensors, cameras in general) is not a major obstacle for this 
application because only a limited number of points must be 
20 measured. These points are often characterized by using 
visible markers. 

Another application of stereovision is stereoscopic 3D 
display, instead of determining the 3D coordinates of some 
points Of an object in a 3D space, it simply needs to display 
25 a paxr of stereoscopic images on a monitor (TV or computer 
monitor) so that the 3D perspective of an image can be seen 
One possible configuration is to capture a pair of images 
usxng two cameras, which observe the parallax effect of an 
object. Then the left eye will view one image of this pair of 
30 stereoscopic images and the right eye will view the other 
The human brain can easily merge this pair of images so that 
the object is viewed as a 3D image. 

The existing 3D digitizing systems and optical motion 
capture systems are, in general, complex and too expensive 
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for the Internet and mass consumer applications. Most of 
these systems incorporate sophisticated optical, electro- 
optical, mechanical and electronic components. Special 
expertise is needed to operate such a digitizer. In addition, 
the existing systems support separately the 3D digitizing and 
motion capture functions. 

SUMMARY 

An object of the invention is to provide a digitizer 
combining functions of capturing stereoscopic images, color 
3D digitizing, and motion capture. 

Another object of the invention is to provide a system 
using the digitizer, which is simple in construction, simple 
to use and affordable for Internet and mass consumer 
applications like conference via Internet, 3D Web, e- 
commerce, off-line and on-line games and any application 
which requires affordable 3D digitizing and/or motion capture 
solution. 

Another object of the invention is to provide methods 
for digitizing and tracking motion of a target object, which 
are implementable using a personal computer and simple 
lighting and video camera components. 

According to the present invention, there is provided a 
digitizer combining functions of stereovision, color 3D 
digitizing and motion capture of a target object, comprising: 

a first camera; 

a second camera; 

a first projection arrangement having a first light 
projector providing lighting for an active 3D range sensing 
for each of the cameras, and a grating element in front of 
the first light projector for projection of an encoded 
pattern on a surface of the target object; 
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a second projection arrangement having a .econd light 
projector providing lighting for an acquisition of texture 
information of the target object,- 

a base onto which the first and second cameras and the 
first and second projection arrangements are mounted in fixed 
relative positions with respect to one another, the cameras 
having optical axes converging through a single point, one of 
the light projectors having an optical axis intersecting with 
the optical axes o£ the cameras at the single point, and 

a communication port connected to the cameras and the 
light projectors, for reception of control signals setting 
operation of the cameras and the light projectors and 
transmission of video signals from the cameras. 

According to the present invention, there is also 

described and a computer having a port connectable with the 
communication port of the digitizer, functions controlling 
operation of the digitizer by generatina th. , '°"^"3 

far 1-1,. ^ . senerating the control signals 

for he stereovision, color digitizing and motion capture 

through the port and generation of digitized data as a result 

Of the processing. 

According to the present invention, there is al.o 
provided a method for diaitizin^ . 
25 steps Of: ^.gxtxzxng a target object, comprising 

capturing basic images of the object with first and 
second cameras without additional iii.„..ation of th: obje 
the cameras having optical axes converging through a s ngle 
poxnt and being aligned in angled directions with respect to 
30 each other so that- i-h^ respect to 

Significant , " ^i-" »"in9 

significant overlapping portions over a depth of measurement 

including the single point; 

patte^T"""' - --^^^ 

pattern is projected; 
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capturing structured images with the cameras; 
illuminating the object with light deprived of a 

patterns- 
capturing texture images with the cameras; 
5 identifying elements of the encoded pattern in the 

structured images; 

determining a position of the elements to produce a set 

of measured points; 

determining 3D coordinates of the measured points using 
10 calibration information in respect with position and 

alignment of the cameras; 

determining coordinates corresponding to each measured 

point in the texture images to produce a digitized image of 

the object . 

15 According to the present invention, there is also 

provided a method for tracking motion of a target object, 
comprising steps of : 

capturing in parallel sequences of images of the target 
object with first and second cameras having optical axes 

20 converging through a single point and being aligned in angled 
directions with respect to each other so that the cameras 
have fields of view having significant overlapping portions 
over a depth of measurement including the single point; 

detecting control points in a first image of each 

25 sequence; 

tracking the control points in the sequences of images; 

determining disparities between the control points in 
the images from the first camera and the images from the 
second camera; 

30 determining 3D positions of the control . points in 

corresponding ones of the images taken at a same time by the 
first and second cameras by using the disparities and 
calibration information comprising relative position and 
angular alignment of the cameras; and 
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generating trajectories of the control points as 
sequences of the 3D positions of the control points 
respectively. 

Preferably, the system of the invention incorporates 
5 elements to capture and transfer a pair of stereo images, to 
obtain 3D coordinates and the color texture of a surface, and 
to capture the displacement of a number of given points in a 
real or quasi-real time. The data capturing process is 
Simplified to make the operation of the digitizer as 
automatic as possible, with the system according to " the 
invention, the 3D model can be first created with some 
Identifiable control points located on the model surface 
Then the 3D position of these control points can be captured 
xn real or quasi-real time, so that the whole model can be 
15 controlled or aninated. 

The oameree oan observe disparity created by the active 
U9ht projector. There is also disparity in a pair o£ images 
captured by the two cameras. This digitizer can be connected 
to a computer via a digital port li.e a USB port, or other 
20 standard high-speed connections. The computer controls the 
cameras and respective light projectors. A snapshot button 
can be used to launch a 3D measurement process. The digitizer 
can be mounted onto a rotational table, „ith the computer 
directly controlling the rotation of the rotational table. It 
i. also possible to place the target object on the rotational 
table so that the angular position of the rotated object can 
be known. 

Preferably, the system provides at least three 
functions. 

30 First, a pair of cameras can capture a pair of 

stereoscopic, images at video rate. The stereo 3D image can be 
created when these two images are displayed on a monitor, 
whach sends one image to the left eye and another image to 
the right eye. These images can be transferred via a high- 
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speed link (Ethernet, Tl, T3, for example) to another 
computer . 

Second, combining the light projectors and both cameras, 
the digitizer provides measurement of the 3D coordinates of a 
5 surface with texture information. The encoded pattern is 
projected on the surface of an object by a light projector 
and both cameras capture the scene. With the surface relief 
of the object, the projected pattern is deformed from the 
point of view of the camera. With a careful calibration 

10 technique, it is possible to determine the 3D coordinates of 
some points on this surface by measuring the deformation of 
the projected pattern. In principle, a combination of one 
camera and one light projector can carry out the measurement 
of the 3D coordinates. The use of two or more cameras, which 

15 cover a common space, combined with one light projector, 
provides three major advantages. First, the weighted average 
values of the 3D measurements obtained by each of the cameras 
correspond to a better 3D measurement.. Second, this 
configuration overcomes more problems caused by an occlusion 

20 effect. Third, the two cameras observe the projected pattern 
from different views so that a better interpretation of the 
deformation of the projected pattern on a discontinued 
surface can be obtained. The measured 3D coordinates can be 
transferred via a high-speed link to another computer, 

25 The third function is to make a motion capture of a 

limited number of points in 3D space. When using some markers 
on a surface, it becomes relatively easy to determine the 3D 
positions of these points. It is also possible to use some 
known points on the object, like the features of the skin, 

30 lips, eyelids, eyes, etc. Of course, the 3D space observed by 

the cameras must be calibrated and a disparity of a given 

point captured by the cameras can be evaluated and its 3D 

position can be calculated. When the number of points to be 

measured is low, it is even possible to determine the 3D 

7 
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positions of these points several times per second. This data 
can be used to control the motion of an object or model. This 
data can be transferred via a high-speed link to another 

computer. 

5 Motion tracking, which analyzes the dynamic motion of a 

subject in a scene captured by any one or both ' cameras 
mounted onto a rotational table, can be achieved with the 
system according to the invention. A servo control may be 
. used. to control (in real time) the rotation of the rotational 
10 table in such a way that the digitizer can follow the dynamic 
motion of the subject. A user, either present or at a 
distance, can also send a command to the computer in order to 
orient the digitizer to a desired direction. Since the servo 
control provides the exact position of the rotational table 
15 xt is evident that the whole 3D space covered by thl 
dxgxtizer mounted on the rotational table is calibrated as a 
known geometric space with respect to the digitizer. This 
function provides the possibility to cover a larger space to 
perform the three basic functions of this apparatus. 
20 instead of mounting the digitizer on the rotational 

table, sometimes, it is convenient to place an object on this 
table. This configuration simplifies the operation to merge 
several views of a 3D object to create a complete 3D model A 
3D digitizer can measure the 3D surface of one single view of 
!5 an object, in order to create a complete object model, it is 
necessary to capture different views of an object. When the 
object is placed on the controlled rotational table, the 
precise position of each view with respect to the other views 
IS known, so it is easy to register several views in a common 
0 3D coordinate system and to merge them to create a complete 
3D model . 
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BRIEF DESCRIPTION OF THE DRAWINGS 

A detailed description of preferred embodiments will be 
given herein below with reference to the following drawings, 
in which like numbers refer to like elements: 

Figure 1 is a schematic diagram illustrating a possible 
working environment of a digitizing system according to the 
invention. 

Figure 2A is a schematic diagram illustrating a front 
view of a digitizer according to the invention. 

Figure 2B is a schematic block diagram illustrating a 
simplified inside view of a digitizer according to the 
invention. 

Figure 3 is a schematic diagram illustrating a 
digitizing system with a digitizer mounted on a rotational 
table 32 according to the invention. 

Figure 4 is a schematic diagram illustrating a 
digitizing system with a target object placed on a rotational 
table according to the invention. 

Figure 5 is a schematic diagram illustrating a possible 
pattern used in a digitizer according to the invention. 

Figure 6 is a schematic diagram illustrating a possible 
control system of a digitizer according to the invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Referring to Figure 2B, there is shown an embodiment of 
a digitizer 1 combining functions of stereovision, color 3D 
digitizing and motion capture of a target object, for example 
an article 3 5 as shown in Figure 4 or a human subject 17 as 
shown in Figure 1. The digitizer 1 has a first projection 
arrangement formed of a first white light projector 3 
optically coupled with a grating element 5 in front of the 
projector 3 for projection of an encoded pattern in the light 
provided preferably through a projection lens 4 and directed 
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on the target object. The digitizer l has a second projection 
arrangement formed of a second white light projector 6 
providing lighting without pattern preferably through another 
projection lens 7. A first color camera 8 preferably with its 
5 lens 9 and a second color camera 10 preferably with its lens 
11 are arranged preferably next to the projection 
arrangements. The light projectors 3, 6 can be operated using 
respective lamp driver boards 12. 13 which are connected to a 
main control board 14 with a peripheral interface 15, e g a 
10 USB port. The main control board 14 and the cameras 8, lo are 
linked respectively to the port 15 so that signals are 
communicated to and from a computer 2 by way of a cable' 16 
The various components of the digitizer i are preferably 
enclosed in a casing 18. 

15 Referring to Figure 2A, the projection arrangements with 

the projection lenses 4,7 are preferably positioned 
vertically one above the other and aligned in angled 
directions with respect to each other so that the two 
projected light fields cover each other at a desired distance 
20 of measurement. Likewise, the cameras with the projection 
lenses 9, li are preferably positioned horizontally and 
spaced from each other. As a result, the camera and 
projection arrangements are positioned symmetrically with 
each other in a cross-like arrangement. 
25 Referring to Figure 2B, the grating element 5 

controllably produces a fringe pattern 3 6 such as the one 
Illustrated in Figure 5, in the light projected by the 
projection lens 4. The cameras 8, lo are positioned aside 
from the projection lenses 4 and 7 and aligned in angled 
.0 directions with respect to each other so that the cameras 8, 
10 have fields of view 19, 20 (see Figure i) directed on the 
xlluminated surface of the object and maximizing the ' 
overlapping with each other over the depth of measurement of 
the object. The video signals produced on the respective 
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video outputs of the cameras 8, 10 represent complementary 
images of the object with a common image portion as a result 
of the fields of view 19, 20 being overlapping. The cameras 
8, 10 may be CCD cameras having respective video outputs. that 
are connected to the port 15 preferably accessible on a rear 
side of the casing 18. 

Referring to Figure 5, the encoded projected pattern 36 
consists of a 2D graphic pattern. This pattern is a sequence 
of color lines. The preferred pattern is a repeating sequence 
of white, black, yellow, black, green, black, cyan and black. 
The sequence is for example repeated 10 times. A magenta 
reference fringe 37 replaces the last black line in the 5th 
sequence. The orientation of the lines can be vertical, 
horizontal or any desired angle. The width of the lines can 
be constant everywhere on the pattern. They can also be 
variable. The magenta line serves as an identifiable 
reference fringe having an absolute 3D position determinable 
in the images captured by the cameras 8, 10, using 
experimentally predefined functions. The reference fringe 37 
is preferably positioned so that each one of the cameras 8, 
10 captures it over the entirety of the object 17, 35 (see 
Figures 1 and 4) , 

Referring to Figure 2B, the optical axis of the 
projection lens 4 should be perpendicular to the surface of 
the projected pattern. The aperture of this lens 4 and its 
focus should be well adjusted so that the projected pattern 
will be sharp enough over all the depth of measurement. 

The optical axes of camera lenses 9, 11 intersect with 

the optical axes of the projection lenses 4, 7 at identical 

distances from the apparatus. These four optical axes 

intersect at the same point, which is usually referred as the 

optical center of the apparatus. The angle between the 

optical axis of the camera 8 and the optical axis of the 

projection lens 4 determines the sensitivity of the distance 

11 
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measurement, referred to as depth, in the direction of the 
optical axis of the projection lens 4. The bigger the angle 
xs, the more sensitive the depth measurement will be. On the 
other hand, the angle should not be too big, in order to keep 
^ an overlapped area over the depth of measurement covered by 
the two cameras 8, lo at least equal to 80-90% of the field 
Of View 13, 20 Of each camera 8, lo. As the projected pattern 
36 xs a repeating pattern of color lines, it is impossible to 
differentiate between lines of the same color. Therefore the 
measurement is without ambiguity only over a certain depth of 
measurement. The bigger the angle between the optical axes of 
the projection arrangement and the camera 8, lo is the 
smaller is the depth of non- ambiguity. 

once the digitizer i is assembled, the optical and 
mechanical configuration of the system is fixed, m order to 
acquire 3D data, the system is calibrated. In the process of 
calibration, the relationship between the position of the 
elements of the projected pattern 36 and their distance to 
the measurement system is determined. 

Referring to Figure S, the cameras 8, 10, the light 
projectors 3, s and the main control board 14 of the 
digitizer l can be powered by a power supply with, for 
example, three outputs 21, 22, 23 providing appropriate 
voltage levels, the power supply receiving energy through a 
connector 24 preferably positioned on the rear side of the 
casing 18. 

, R.f«ri„g to Figures 1 and a sy.t.„ using th. 

digitizer 1 according to the present invention is provided 
wxth a c=„puter 2 with a port 38 for acquiring the video 
=.gnal3 coding froo, the cameras 8, io and to send cc^ands 
from the computer 2 to the main control board 14 The 
computer 2 will process the video signals coming £rom the 
cameras 8, 10. The computer 2 will also initiate , 
measurement se<^ence and control the two light projectors 3 
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6 in a way that images with a projected pattern 36, without 
pattern and with uniform white illumination are acquired. The 
acquisition and projector control are preferably controlled 
via the main control board 14 by the computer 2 through a 
5 cable 16 connected to the port 15. 

Preferably, the measurement process for the 3D 
digitizing is as follows. Images are first acquired with the 
cameras 8, 10 without illumination. The light projector 3 is 
then energized in order to generate a pattern projected on 

10 the object 17. Images are acquired with the cameras 8, 10 
with the projected pattern. The light source of the 
projection arrangement 3 is then deenergized and the light 
source of the projection arrangement .6 is energized in order 
to generate a uniform white light illumination of the 

15 measurement space. Images are acquired with the cameras 8, 10 
with uniform white illumination. Elements of the projected 
pattern are identified in the images of the object to be 
measured with the projected pattern. The position of the 
elements of the projected pattern is determined in the images 

20 of the object to be measured with the projected pattern. This 
step generates a set of points measured. The 3D coordinates 
of all measured points are determined using the calibration 
information. The coordinates corresponding to. each measured 
point in the texture images acquired with the uniform white 

25 light illumination are further determined. 

For every recording sequence, a set of video images is 
processed into a suitable software application to retrieve 
the 3D information from the structured images, along with the 
texture information. Different algorithms can be used for 

30 retrieving the shape and texture information from the video 
images using projected structured light. Most of these 
algorithms are based on the identification and localization 
of the fringes. 
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The first step of these algorithms will be to detect the 
projected fringes in the acquired images. Image processing 
algorithms are used to segment the fringes, which permits to 
determine for each pixel of the image of which fringe the 
pixel is a part of. Once the fringes are segmented, the 
position of all fringes in all lines of the image can be 
determined by finding the maximum intensity of the fringe or 
by calculating the center of gravity of the intensity 
distribution of a fringe. If colored fringes are used, the 
color of each fringe is determined. 

The reference fringe 37 (see Figure 5) encoded into the 
projected pattern is made uniquely identifiable, preferably 
by its color. The next step in the 3D digitizing process is 
the segmentation and localization of the reference fringe 37 
in the acquired images. For most lines, the position of the 
reference fringe is then known. For each line of the image, 
the fringes are numbered corresponding to their color 
beginning from the reference fringe to the- edges of the 
image. The color permits to number the fringes correctly as 
long as the object contains no steps that are larger than the 
depth of non-ambiguity. The positions of the identified 
fringes are then converted from real world coordinates into 
real world units, like millimeters for example. For each 
camera 8, lO in conjunction with the light projector 3 a 
calibration table is created in a separate calibration 
process. This calibration table permits to convert the 
measured positions of the identified fringes in 3D 
coordinates in geometric units. 

The conversion function can be: 
(x,y,z)^ = /(u,j,n) 

where: (x,y.z)^ are real world coordinates; 
n is a number of the identified fringe; 
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j is a line of the image in which the fringe n is 
identified; and 

u is a position of fringe n in line j . 

Once a point cloud is generated based on all the 3D 
5 coordinate measurements from a set of images acquired by one 
of the cameras 8, 10, for each point of the point cloud a 
color texture vector is determined calculated from the 
coordinates and the texture image acquired with uniform white 
light illumination. 

10 The computer 2 may be used to determine the color 

detected in each pixel of the image obtained with white light 
projection and, for each pixel, to determine the ratio of 
each of the red, green and blue color channels to the maximum 
of the three channels. Color enhancing can then be achieved 

15 by applying the inverse of these ratios to the corresponding 
pixels in the image obtained with the structured light 
projection. 

This process of 3D digitizing is executed for the camera 
8 and the camera 10 generating two independent 3D models. In 

20 a final step, these two models can be merged. The relative 
position of the models is well known from the calibration 
process so the merging of the models and textures can be 
achieved by using an algorithm of cylindrical projection or 
cubical projection. 

25 A single full digitized image of the object is thus 

obtained as a function of the spatial relationship between 
the real 3D position of the detected fringe maxima in each 
line of the image with structured light projection and the 
real 3D position of the structured light projection, the 

30 texture image being integrated into the digitized image. 

Referring to Figure 4, when the object to be measured is 
placed on a rotational table 32 connected to the computer 2 
through a cable 34 for the purpose of capturing multiple 
views, the computer 2 controls precisely the position of the 
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rotation so that the relative position between each view is 
known. It is very straightforward to put multiple partial 
models together to create a complete model. 

Referring to Figure 6, the main control board 14 
receives commands from and transmits signals to the computer 
2 via the cable 16. in the illustrated embodiment, the main 
control board 14 has a CPU 26, a USB interface 27, RAM and 
PROM memories 28 for data and program and an Input/Output 
unit 29. The commands that are recognized by the board are, 
for example: resetting the digitizer 1; turning the light of 
the light projector 3 on and off; turning the light of the 
light projector 6 on and off; and starting the digitizing 
process . 

The lamp driver boards 12, 13 receive on/off signals via 
the main control board 14. The intensity of the light 
projector 3, 6 can be manually adjusted with potentiometers 
30, 31. 

Referring to Figures 1 and 2, to perform stereovision 
display, the images captured by the two cameras 8 and 10 are 
first transferred to the host computer 2, then a stereo pair 
of images is calculated, the images being preferably 
compressed before being sent to another computer (not shown 
in the Figures) . The images received by the second computer 
will be decompressed if need be and displayed on a monitor 
screen 33. In order to see a 3D stereo image, one of the two 
images need to be seen by the left eye and the other image 
seen by the right eye. This can be accomplished by 
synchronizing the display of the two images using LCD shutter 
eyeglasses (not shown in the Figures) . 

For motion capture function, the disparity observed by 
the cameras 8, 10 is calibrated in a common 3D space of these 
cameras 8, 10. The 3D position of a corresponding point 
captured by two cameras can then be determined. A method for 
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real time disparity measurement is developed to ensure a 
motion capture of limited points at quasi -video rate. 

Referring to Figure 3, an object in motion can be 
tracked. The rotational table 32 controlled by the computer 2 
5 is then used . in order to make the digitizer 1 follow the 
ob j ect in motion . The apparatus has a limited measurement 
space. In order to keep the object in the center of the 
measurement space, images are acquired by the cameras 8, 10 
and transferred to the computer 2. The difference between 

10 subsequent video images is determined and used to calculate 
the displacement of the object. The computer 2 controls the 
rotational table 32 in order to keep the object in the center 
of the measurement space. 

In the motion capture mode, the computer 2 may operate 

15 so that sequences of images in the video signals from the 
first and second cameras 8, 10 are acquired in parallel. 
Control points in a first image of each sequence are then 
detected and tracked in the sequences of images while 
disparities between the control points in the images from tjie 

20 camera 8 and the images from the camera 10 are. determined. 3D 
positions of the control points in corresponding ones of the 
images from the cameras 8, 10 taken at a same time are also 
determined by using the disparities and calibration 
information involving relative position and angular alignment 

25 of the cameras. Trajectories of the control points can be 
generated as sequences of the 3D positions of the control 
points respectively. The control points may be formed for 
example of specific markers positioned on the object or 
points in the images of the object with high derivatives in 

30 intensity. 

The computer 2 may have compressing and decompressing 
functions applicable on the digitized data, and functions for 
transmitting the digitized data through a communication port 
of the computer 2 . 
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Although white light is preferred for the projectors 3. 
6, other colors may possibly be used depending on the needs. 
Aperture and shutter mechanisms (not illustrated in the 
Figures) can be used with the projectors 3, 6. The cameras 8, 
10 used in the digitizer 1 may have adjustable gain features 
if desired. 

While embodiments of this invention have been 
illustrated in the accompanying drawings and described above, 
it will be evident to those skilled in the art that changes 
and modifications may be made therein without departing from 
the essence of this invention. All such modifications or 
variations are believed to be within the scope of the 
invention as defined by the claims appended hereto. 
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CLAIMS : 

1. A digitizer combining functions of stereovision, 
color 3D digitizing and motion capture of a target object, 
comprising: 

a first camera; 
a second camera; 

a first projection arrangement having a first light 
projector providing lighting for an active 3D range sensing 
for each of the cameras, and a grating element in front of 
the first light projector for projection of an encoded 
pattern on a surface of the target object; 

a second projection arrangement having a second light 
projector providing lighting for an acquisition of texture 
information of the target object; 

a base onto which the first and second cameras and the 
first and second projection arrangements are mounted in fixed 
relative positions with respect to one another, the cameras 
having optical axes converging through a single point, one of 
the light projectors having an optical axis intersecting with 
the optical axes of the cameras at the single point; and 

a communication port connected to the cameras and the 
light projectors, for reception of control signals setting 
operation of the cameras and the light projectors and 
transmission of video signals from the cameras, 

2. The digitizer according to claim 1, wherein the first 
and second projection arrangements are aligned in angled 
directions with respect to each other so that projected light 
fields cover each other at a predetermined measurement 
distance from the projection arrangements, and the first and 
second cameras are aligned in angled directions with respect 
to each other so that the cameras have fields of view having 

19 
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significant overlapping portions over a depth of measurement 
around the measurement distance. 

3. The digitizer according to claim 2, wherein: 
the light projectors are white light projectors; 
the cameras are color cameras; and 

the encoded pattern comprises a repetition of a sequence 
of color lines, one of which being used as an identifiable 
reference having an absolute 3D position determinable in 
xmages captured by the cameras with the lighting provided by 
the f.rst projection arrangement using experimentally 
predefined functions. 
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4. The digitizer according to claim 3, wherein the 
sequence of color lines comprises white, black, yellow 
black, green, black, cyan and black, the black line closest 
to an optical axis of the first projection arrangement being 
replaced by a magenta reference line. 

5. The digitizer according to claim l, wherein the first 
projection arrangement has a projection lens optically 
coupled with the first light projector and having an optical 
axis substantially perpendicular to a surface of the encoded 
pattern. 

6. The digitizer according to claim 2, wherein the other 
one of the first and second light projectors has an optical 
axxs intersecting with the optical axes of the cameras at the 
single point, the projection arrangements and the cameras 
bexng positioned symmetrically with each other in a cross- 
like arrangement. 

7. The digitizer according to claim 6, wherein the first 
projection arrangement has a projection lens optically 
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coupled with the first light projector, the optical axis of 
the first camera having an angle with a projection axis of 
the projection lens, the angle being adjusted so that the 
overlapping portions equal or exceed 80% of the field of view 
5 of each camera. 

8. The digitizer according to claim 1, wherein the 
cameras turn into action and the communication port transfers 
stereoscopic images captured by the cameras at video rate in 

10 response to control data in the control signals associated 
with the function of stereovision. 

9. The digitizer according to claim 1, wherein the first 
light projector turns on and the communication port transfers 

15 images captured by the cameras in response to control data in 
the control signals associated with the function of color 3D 
digitizing, 

10. The digitizer according to claim 1, wherein the 
20 communication port transfers images captured by the cameras 

in response to control data in the control signals associated 
with the function of motion capture. 

11. The digitizer according to claim 1, further 
25 comprising lamp driver boards connected to and controlling 

respective ones of the light projectors, and a main control 
board coupled between the communication port and the lamp 
driver boards. 

30 12. The digitizer according to claim 1, further 

comprising a casing enclosing the cameras and the projection 
arrangements, the base being an integral portion of the 
casing. 
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13. A digitizing system comprising: 

a digitizer combining functions of stereovision, color 
3D digitizing and motion capture of a target object, 
comprising: 

a first camera; 
a second camera; 

a first projection arrangement having a first light 
projector providing lighting for an active 3D range 
sensing for each of the cameras, and a grating element 
m front of the first light projector for projection of 
an encoded pattern on a surface of the target object; 

a second projection arrangement having a selond 
Ixght projector providing lighting for an acquisition of 
texture information of the target object; 

a base onto which the first and second cameras and 
the f.rst and second projection arrangements are mounted 
xn fixed relative positions with respect to one another 
the cameras having optical axes converging through a 
sxngle point, one of the light projectors having an 
optical axis intersecting with the optical axes of the 
cameras at the single point; and 

a communication port connected to the cameras and 
the light projectors, for reception of control signals 
setting operation of the cameras and the light 
projectors and transmission of video signals from the 
cameras; and 

a computer having a port connectable with the 
communication port of the digitizer, functions controlling 
operation of the digitizer by generating the control signals 
for the stereovision, color 3D digitizing and motion capture 
and functions for a processing of the video signals received 
through the port and generation of digitized data as a result 
of the processing. 
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14. The digitizing system according to claim 13, wherein 
the functions for the color 3D digitizing comprise a 
modelling process causing the computer to successively: 

acquire basic images in the video signals from the 
5 cameras without illumination of the object from the 
projectors; 

acquire structured images in the video signals from the 
cameras with illumination of the object from the first 
projector; 

10 acquire texture images in the video signals from the 

cameras with illumination of the object from the second 
projector; 

identify, elements of the encoded pattern in the 
structured images; 
15 determine a position of the elements to produce a set of 

measured points; 

determine 3D coordinates of the measured points .using 
calibration information stored in the computer; and 

determine coordinates corresponding to each measured 
20 point in the texture images to produce a digitized image of 
the object, the digitized data comprising the digitized image 
of the object. 

15. The digitizing system according to claim 14, 
25 wherein: 

the elements are identified by detecting projected 
fringes in the structured images, the projected fringes 
forming the elements of the encoded pattern identified in the 
structured images, the projected fringes being segmented; 
30 the position of the elements is determined by 

determining a relative position of the fringes in lines of 
the structured images by finding a maximum intensity or 
calculating a center of gravity of an intensity distribution 
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of the fringes, the position of the fringes forming the 
measured points; and 

the 3D coordinates are determined by locating a 
reference fringe in the structured images based on a 
distinctive one of features thereof, numbering the fringes 
according to the feature thereof beginning from the reference 
frxnge to edges of the structured images, converting the 
relative position of the fringes into real world coordinates 
xn real world units with respect to the reference fringe, and 
converting the position in real world coordinates into the 3D 
coordinates in geometric units using a calibration table 
created ^n a separate calibration operation for each one of 
the cameras, the calibration tables forming the calibration 
information. 

16. The digitizing system according to claim 15 
wherein : 

the distinctive one of the features is a color, the 
fringes being numbered according to the color thereof; 
the converting uses a function: 

ix,y,zY =f{u,j,ri) 
where : 

{x,y,2y i.a the position in real world coordinates, 
n is a fringe number, 

j is a line number in the image for the fringe n, 

and 

u is the position of the • fringe n in the line j - 

and ' 

the coordinates in the texture images are determined by 
generating a point cloud based on the 3D coordinates from a 
set of the images acquired by one of the cameras, and 
determining a color texture vector from the coordinates and 
the texture images for each point of the point cloud 
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17. The digitizing system according to claim 16, wherein 
the computer further determines for each point of the point 
cloud a ratio of each red, green and blue color channels to a 

5 maximum of the channels, and applies an inverse of the ratios 
to corresponding points in the structured image. 

18. The digitizing, system according to claim 16, wherein 
the modelling process is performed separately on the images 

10 from the first and the second cameras respectively to produce 
two independent 3D models which are later merged together, 

19. The digitizing system according to claim 13, further 
comprising a rotational table controlled by the computer for 

15 rotation of one of the digitizer and the object for creation 
of a textured 3D model of the object, the digitized data 
comprising the textured 3D model of the object. 

20. The digitizing system according to claim 13, wherein 
20 the functions for the stereovision cause the computer to 

calculate a stereo pair of the images for display with a 3D 
impression of a scene, the digitized data comprising the 
stereo pair of the images. 

25 21. The digitizing system according to claim 13, wherein 

the functions for the motion capture cause the computer to: 

acquire in parallel sequences of images in the video 
signals from the first and second cameras; 

detect control points in a first image of each sequence; 
30 track the control points in the sequences of images; 

determine disparities between the control points in the 
images from the first camera and the images from the second 
camera; 
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determine 3D positions of the control points in 
corresponding ones of the images from the first and second 
cameras taken at a same time by using the disparities and 
calibration information comprising relative position and 
angular alignment of the cameras; and 

generate trajectories of the control points as sequences 
Of the 3D positions of the control points respectively, the 
digitized data comprising the 3D positions and the 
trajectories of the control points. 

22. The digitizing system according to claim 21, wherein 
the control points comprise specific markers positioned on 
the object, 

23. The digitizing system according to claim 21, wherein 
the control points comprise points in the images of the 
object with high derivatives in intensity. 



24. The digitizing system according to claim 13, wherein 
the computer further has compressing and decompressing 
functions applicable on the digitized data 



25. The digitizing system according to claim 13, wherein 
the computer further has functions for transmitting the 

25 digitized data through a communication port of the computer. 

26. A method for digitizing a target object, comprising 
steps of: 

capturing basic images of the object with first and 
second cameras without additional illumination of the object 
the cameras having optical axes converging through a single 
point and being aligned in angled directions with respect to 
each other so that the cameras have fields of view having 
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significant overlapping portions over a depth of measurement 
including the single point; 

illuminating the object with light in which an encoded 
pattern is projected; 
5 capturing structured images with the cameras; 

illuminating the object with light deprived of a 
pattern; 

capturing texture images with the cameras; 

identifying elements of the encoded pattern in the 
10 structured images; 

determining a position of the elements to produce a set 
of measured points; 

determining 3D coordinates of the measured points using 
calibration information in respect with position and 
15 alignment of the cameras; 

determining coordinates corresponding to each measured 
point in the texture images to produce a digitized image of 
the object. 

20 27. The method according to claim 2S, wherein: 

the elements are identified by detecting projected 
fringes in the structured images, the projected fringes 
forming the elements of the encoded pattern identified in the 
structured images, the projected fringes being segmented; 

25 the position of the elements is determined by 

determining a relative position of the fringes in lines of 
the structured images by finding a maximum intensity or 
calculating a center of gravity of an intensity distribution 
of the fringes, the position of the fringes forming the 

30 measured points; and 

the 3D coordinates are determined by locating a 
reference fringe in the structured images based on a 
distinctive one of features thereof, numbering the fringes 
according to the feature thereof beginning from the reference 
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fringe to edges of the structured images, converting the 
relative position of the fringes into real world coordinates 
.n real world units with respect to the reference fringe, and 
converting the position in real world coordinates into the 3D 
coordinates in geometric units using a calibration table 
created in a separate calibration operation for each one of 
the cameras, the calibration tables forming the calibration 
information. 



28. The method according to claim 27, wherein: 
the distinctive one of the features is a color, the 
fringes being numbered according to the color thereof; 
the converting uses a function: 
(x,y,2)^=/(„,y,„) 
15 • where: 

(x.y,2y is the position in real world coordinates, 
n is a fringe number, 

j is a line number in the image for the fringe n, 

and 

'° " "^^^ position of the fringe n in the line i- 

and 

the coordinates in the texture images are determined by 
generating a point cloud based on the 3D coordinates from a 
set of the images acquired by one of the cameras, and 
25 determining a color texture vector from the coordinates and 
the texture images for each point of the point cloud. 

29. The method according to claim 28, further comprising 
Steps of: 

3 0 for each point of the point cloud, determining a ratio 

Of each red, green and blue color channels to a maximum of 
the channels; and 
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applying an inverse of the ratios to corresponding 
points in the structured image. 

30. A method for tracking motion of a target object, 
5 comprising steps of : 

capturing in parallel sequences of images of the target 
object with first and second cameras having optical axes 
converging through a single point and being aligned in angled 
directions with respect to each other so that the cameras 
10 have fields of view having significant overlapping portions 
over a depth of measurement including the single point; 

detecting control points in a first image of each 
secjuence; 

tracking the control points in the sequences of images; 
15 determining disparities between the control points in 

the images from the first camera and the images from the 
second camera; 

determining 3D positions of the control points in 
corresponding ones of the images taken at a same time by the 
20 first and second cameras by using the disparities and 
calibration information comprising relative position and 
angular alignment of the cameras; and 

generating trajectories of the control points as 
sequences of the 3D positions of the control points 
25 respectively. 

31. The method according to claim 30, wherein the 
control points comprise specific markers positioned on the 
object . 

30 

32. The method according to claim 31, wherein the 
control points comprise points in the images of the object 
with high derivatives in intensity. 



29 



wo 01/81859 



PCT/CAOl/00560 



1/6 




2 



FIG. 1 



wo 01/81859 



PCT/CAOl/00560 



2/6 




wo 01/81859 



PCT/CAOl/00560 



3/6 










ROTATIONAI 


TABLE 





32 




FIG. 3 



wo 01/81859 



PCT/CAOl/00560 





FIG. 4 



wo 01/81859 



PCT/CAOl/00560 




wo 01/81859 



PCT/CAOl/00560 



6/6 




16 



31 




14 



MAIN 
CONTROL 
BOARD — 



LAMP 
DRIVER 
BOARD h 



USB 



26 



CPU 


RAM 
ROM' 


J/Q - 







27 

28 

J 

29 



PROJECTOR 



21 
L 

22 
23 



POWER 
SUPPLY 



LAMP 

DRIVER 

BOARD 



12- 



30 



FIG. 6 



INTERNATIONAL SEARCH REPORT 



Ir llensi Application No 

PCT/CA 01/00560 



Aocofding to Inl emaltonal Patent Classificalion (IPC) or io both natlonaJ ctas&iflcatlon and IPC 
B. RELOS SEARCHED ' ' 



Minlmurn documwtelbn searched (classtflcajlon system rottowed by classincalion symbols) 

IPC 7 GOiB 



Documentation searched other than minimum documentation to the extent that such documents are kicluded in the fields searched 



Electronic data base consulted during the International search (name of data 

EPO-Internal 



base and. where practical, search terms used} 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Categoiy • Citation of document wBh indication, where appropriate, of trie relevant 



US 5 747 822 A (M.J. SIBCLAIR, F.E. VITZ) 
5 May 1998 (1998-05-05) 



* the whole document * 
figures 1-5 

WO 89 09378 A (KREON INGENIERIE MARKETING) 
5 October 1989 (1989-10-05) 

* the whole document * 
figures 1-15 

US 6 028 672 A (ZHENG JASON CENG) 
22 February 2000 (2000-02-22) 

* the whole document * 
figures 1-10 

-/- 



Relevant to dalm Mo. 



"x] Pufther doojments are listed In the continuation of box C. 



0 



1.2,5-9 

13.19, 
20.25,26 



1.2,5,7. 
8,13,26, 
30 



1-3,7,8, 
13,26 



Pale nt remily members are listed in annex 



Spedal categories of dted documents : 

document defining the general slate of the art which (s not 
considered lo be of panicutar relevance 

earter document but published on or after the international 
ruing date 

V docurDent which may throw doubts on priorify dalmfslor 
Which is c«ed to establish the publication date of another 
dtatton or other special reason (as specified) 

•0* document referring to an oral disdosure, use. exhibition or 
other means 

•P* document published prior to the international filing date but 
later than the prtorWy dale claimed 



V later document published aner the Intefnailonal filing date 
or priority date and nol in conflict with the application but 
dted to understand the principle or theory underlying the 
Invention 

'X' document of particular relevance; the daimed Invention 
an not be considered novel or cannot be considered to 
involva an inventive step when the documeni is taken alone 

•Y* document of particular relevance; the claimed Inveniion 
S!1[I2L^ considered JO Invoke an Inventive step when the 
document is combined with one or more other such docu- 
ments, such combination being obvious to a person sMIled 
n the art. 

'V document member of the same patent family 



il completion of the international search 



1 August 2001 



Dale of mailing of the international search report 

09/08/2001 



Name and mailing address of the ISA 



European Patent orrice, P. a 5818 Paientlaan 2 
NL-2280HVnjiswijlc 
TeL (^.31 -70) 340-2040. Tx. 31 651 epo nt. 
Fax: (♦31-70)340-3016 



Authorized officer 



Visser, F 



Peim PCJA3^J2^0 (McortdshMt) (July 1992) 



INTERNATIONAL SEARCH REPORT 



In itlonal Application No 

KCI/CA 01/00560 



C^Contlnuation) DOCUMENTS CONSIDERED TO BE REUEVANT 



Category 



Citation or dooumeni, with Indlcatlon.wnefe appropriala, of the relevant passages 



Relevant to dalm No. 



UO 99 24786 A (J. COHEN-SABBAN) 
20 Hay 1999 (1999-05-20) 
* the whole document * 
figures 1-9 

GB 2 264 602 A (3D SCANNERS LTD.) 

1 September 1993 (1993-09-01) 

page 2, paragraph 6 -page 4, paragraph 2; 

figures 1,2 



1,3-5, 
13,26,30 



1.12 



Form PCr/ISA^lO (contihuaUon d seoand sheet) <July iggS) 



INTERNATIONAL SEARCH REPORT 


In ttlonal Application No 

PCT/CA 01/00560 


Patem document 
cited In search report 


Put>ljcation 
date 


Patent family 
member(s) 


Publication 
date 



us 5747822 A 05-05-1998 NONE 



WO 8909378 A 05-10-1989 FR 2629198 A 

CA 1328913 A 

EP 0348247 A 

FR 2629225 A 

FR 2629233 A 



US 6028672 A 22-02-2000 US 6147760 A 14-11-2000 

WO 9924786 A 20-05-1999 AU 479139rA 31-05-1999 
EP 0950168 A 20-10-1999 

GB 2264602 A 01-09-1993 NONE 



29-09-1989 

26- 04-1994 

27- 12-1989 
29-09-1989 
29-09-1989. 



Fbim PCT/ISM2to tnunilainiy aonea) (JuV i«82) 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the apphcant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



"«S PAGE BLANK (usPTO) 



